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Abstract 

The results of a multi-lepton search conducted by the CMS collaboration with 5 
fb _1 of data collected from 7 TeV pp collisions are used to place the first bound on 
the rare flavor-changing decay of the top quark to a Higgs boson and charm quark. 
Combining results from a number of exclusive three- and four-lepton search channels 
yields an estimated upper limit of Br(t —> ch) < 2.7% for a Higgs boson mass of 125 
GeV. The sensitivity of future dedicated searches for t — > ch could be improved by 
adding exclusive same sign di-lepton channels, as well as by sub-dividing channels 
based on 6-quark tagging and partial kinematic top quark and Higgs boson tagging. 
This bound may be interpreted more widely within a range of new physics processes 
that yield final states with a VF-boson in association with a Higgs boson. For such 
processes with kinematics that are similar to top-anti-top production and decay, 
the estimated limit on cross section times branching ratio corresponds to roughly 
a-Br(pp -> WhX) < 9 pb. 



1 Introduction 



Once the existence of a Standard Model-like Higgs boson is established at the Large Hadron 
Collider (LHC) it will be possible to search for new physics produced in association with the 
Higgs. The myriad decay modes of the Higgs boson offer a wide range of possibilities for such 
searches. One attractive possibility is to focus on the leptonic final states of the Higgs boson 
in association with additional leptons coming from other particles accompanying the Higgs. 
Multi-lepton signatures originating from Standard Model production and decay of the Higgs 
boson itself provide considerable sensitivity p], and in conjunction with additional leptons 
could provide powerful probes of non-Standard Model processes that include a Higgs. 

One class of non-Standard Model processes of interest are those in which the Higgs 
boson appears only rarely in association with other particles. In this case, observation 
of a new physics process requires a large production cross section, making it fruitful to 
consider Standard Model processes with large production cross section. The production of 
top-anti-top quark pairs is particularly attractive in this respect, with a cross section of 
100's of pb at the LHC. This suggests looking for Higgs bosons in the decay products of 
the top quark, such as would arise through the rare neutral flavor- changing transition to 
a charm quark, t — > ch. In fact, no symmetry forbids this decay, but the Standard Model 
contribution to the branching ratio suffers both GIM and second-third generation mixing 
suppression and is extremely small, of order Br(i — > ch) SM ~ 1CT 13 — 1CT 15 El II]- Thus 
a positive observation of the process t — > ch well above the Standard Model rate would be 
a convincing indication of new physics beyond the Standard Model. 

Pair production of top-anti-top followed by the rare decay t — > ch gives rise to multi- 
lepton final states with up to five leptons. The leading processes involve leptonic charged- 
current decay of one of the top quarks, t —> Wb with W — > £is, and flavor- changing decay 
of the other top quark, i — > ch with leptonic final state decay modes of the Higgs boson. 
These include h — > WW* — > £i/£v, and h — > tt with leptonic decay of the tau-leptons, 
r — > £X, as well as h — > ZZ* — > jj££, uuii, ££££. Hadronic decay of one of the top quarks, 
t — > Wb with W — > jj, and flavor-changing decay of the other top quark, i — > ch with 
h — > ZZ* — > ££££ also contributes. Such multi-lepton final states have relatively low 
Standard Model backgrounds, making them promising targets for a multi-lepton search. 

To investigate the utility of searching for t — > ch in this way, we make use of the 
results of a multi-lepton search conducted by the CMS collaboration with 5 fb _1 of data 
collected from 7 TeV pp collisions [3] to estimate a limit on the branching ratio Br(t — > ch). 
The power of this search lies in the combination of numerous exclusive channels. While 
any individual channel alone is not necessarily significant, the exclusive combination across 
multiple channels is found to provide an interesting sensitivity to Br(t — > ch) at the percent- 
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level. To our knowledge this is the first use of the Higgs boson as a probe for new physics 
in existing data. 

The neutral flavor-changing decay of a top quark to the Higgs boson and charm quark, 
t — > ch, is of interest because it provides a direct probe of flavor violating couplings to 
the Higgs sector for the quark that is most strongly coupled to that sector. Previous 
probes of flavor violating couplings to the Higgs sector for the lighter quarks have been 
only indirect. It is also of general interest because up-type quark flavor violation is less 
well constrained than that for down-type quarks. Given that this process has not been 
investigated experimentally at any level previously, the percent-level bound on Br(t — > ch) 
obtained here begins to open up an interesting new window into flavor violating physics. 

In §2 we present an effective operator analysis of the rare decay t — >■ ch and give 
the relation between the branching ratio and new physics scale of the leading operator 
that contributes to this process. In §3 we review multi-channel multi-lepton searches and 
compare the results of a CMS search with our simulation of top quark pair production and 
decay, including t — > ch, to obtain the first limits on Bi(t — > ch) for a Standard Model-like 
Higgs boson. We also suggest improvements that could increase the intrinsic sensitivity of 
future dedicated multi-lepton searches for t — > ch. In §4 we discuss the wider applicability 
of this result couched in terms of a cross section times branching limit on new physics that 
yields final states with a H^-boson in association with a Higgs boson. 

It should be noted that, although throughout we refer to the flavor- violating decay 
of the top quark to a Higgs boson as t — > ch, since the identity of the charm quark is 
not integral to the analysis, the discussion and results apply more generally to the decay 
t — > Xh with inclusive X final states. 

2 Effective Operator Description of t — > ch 

New physics contributions to the flavor-violating top quark decay t — > ch may be encoded 
in an effective field theory description of the operators that can contribute to this process. 
For the field content of the minimal Standard Model, the leading coupling of the Higgs 
boson to up-type quarks is through the renormalizable dimension-four Yukawa coupling 

XijQiHuj + h.c. (1) 

where the quark fields are two component complex Weyl Fermions. The most relevant 
sub-leading interactions coupling the Higgs to up-type quarks come from dimension-six 
operators. Up to operator relations at this order, these may written in terms of a single 
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non-renormalizable operator 



^H^HQiHuj + h.c. 



M 2 



(2) 



At this order in an effective field theory expansion, both the Yukawa coupling ([T]) and 
dimension- six operator (j2j) contribute to the up-quark mass matrix and effective coupling 
to the physical Higgs boson 

rriijUiUj + XijhuiUj + h.c. (3) 
where the up-quark mass matrix is given by 



rrii 



v 

71 



Xij + 2M 2 ^ ij 



v 
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= 71 ij 
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and where H° = A=(v + h) and XJ- is the mass effective Yukawa coupling. The Higgs 
effective Yukawa coupling, A^-, of the physical Higgs boson to up-quarks is given at this 
order in the effective field theory description by the derivative of the mass matrix with 
respect to the Higgs expectation value 
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Since the mass effective Yukawa is by definition diagonal in the mass basis, flavor- violating 
interactions come only from the second term in parentheses in (151). The off-diagonal top- 
charm coupling to the Higgs boson in the effective field theory description up to dimension- 
six is then 



tc 



V2M 2 



(6) 



Misalignment between the mass and Higgs effective Yukawa couplings, A™ and A^-, vanishes 
in the M — > oo limit. 

The partial decay width of the top quark to a Higgs boson and massless charm quark 
from the effective Higgs interaction (J3]) with the coupling (JSJ) is given by 



r(t ->■ ch) 
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where Gp 1 



y/2v 2 . For comparison, the partial decay width of the top quark to the 



VT-boson and massless 6-quark through the minimal charged current interaction is 



r(t ->■ Wb) 
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Assuming Br(£ — > Wb) is close to unity, the leading order branching ratio for t — > ch is 
then given by 



For Higgs boson and top quark masses of = 125 GeV and m t = 173.5 GeV respectively, 
the numerical value of the branching ratio in terms of the dimension-six operator scale and 
flavor-violating Higgs effective Yukawa coupling are 



3 A multi-lepton search for t — > ch 

Multilepton searches at hadron colliders provide great sensitivity to new physics processes. 
In this work we follow and use the results of the multi-lepton search strategies adopted by 
the CMS collaboration |5j |6]. The sensitivity to new physics arises from dividing three- 
or more-lepton final states into a large number of exclusive search channels based on lep- 
ton flavor and charge combinations, hadronic activity, missing transverse energy, and the 
kinematic properties of the leptons in an event. We first review the details of this search 
strategy before applying it to obtain a bound on Br(£ — > ch). 

3.1 Multi-lepton signal channels 

Standard Model backgrounds to multi-lepton searches for new physics are small and may 
be further reduced by imposing cuts on hadronic activity or missing energy. In this case 
hadronic activity is characterized by the variable Ht, the scalar sum of the transverse jet 
energies for all jets passing the preselection cuts. The missing transverse energy, MET, 
is given by the magnitude of the vector sum of the momenta of all reconstructed objects. 
Both Ht and MET are sensitive discriminating observables for new physics in a given 
lepton flavor and charge channel. 

The CMS multilepton search [5] exploits the background discrimination of Ht and MET 
in the following way: Events with H T > 200 (MET > 50) GeV are assigned HIGH H T 
(MET), while those with H T < 200 (MET < 50) GeV are assigned LOW H T (MET). 
The HIGH Ht and HIGH MET requirements (individually or in combination) lead to a 
significant reduction in Standard Model backgrounds^ 

1 It is also possible to reduce backgrounds using an St variable defined to be the scalar sum of MET, 
Ht, and leptonic pt [5], but for simplicity we will not make use of St here. 



Br(t -»■ ch) 



(9) 



4V2G 3 F m 2 t M±\V tb \ 2 (1 - m 2 w /m 2 t f{\ + 2m 2 v /m 2 ) 




(10) 
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Further background reduction may be accomplished with a Z-boson veto, in which the 
invariant mass of opposite-sign same-flavor (OSSF) lepton pairs is required to lie outside 
a 75 — 105 GeV window around the Z mass; we simply denote events passing the Z veto 
as No Z . In the case of 3£ events, it is also useful to differentiate between events with no 
OSSF pairs, which we denote DYO to indicate no possible Drell-Yan pairs, and one OSSF 
pair which we denote DY1. Although the CMS multi- lepton analysis [5J E] also includes 
channels with one or more objects consistent with hadronically decaying r-leptons, in this 
analysis we will focus our attention on £ = e, /i only. We do implicitly include leptonically 
decaying r-leptons in our analysis, which for all practical purposes in the detector are 
simply e- or /i-leptons. 

The 3£ or 4£ channels may be divided into 20 possible combinations of HIGH/LOW; 
MET HIGH/LOW; Z /No Z; and DY0/DY1. The 20 channels are presented in Tabled For 
each of the 3£ and 4£ categories, channels are listed from top to bottom in approximately 
descending order of backgrounds, or equivalently ascending order of sensitivity, with the 
last such channel at the bottom dominated by Standard Model backgrounds. However, all 
channels contribute to the limit. 

3.2 Simulation details 

We closely follow the CMS multilepton analysis [5], applying the same cuts to our signal 
sample and making use of the CMS background estimates and observations with 5 fb _1 
of 7 TeV pp collision data. For our signal, we simulate ti production events with one 
side decaying through conventional charged current interaction via t — > Wb and the other 
side decaying via t — > ch. For definiteness we take = 125 GeV with Standard Model 
branching ratios. For simulating signal processes, we have used MadGraph v4 [7J [5] and 
rescaled the ti production cross section to the NLO value a(pp — > ti) = 165 pb at 7 
TeV [9]. The Higgs boson was decayed inclusively using BRIDGE [TU]. The branching 
ratios and total width for Higgs decay in BRIDGE were taken from the LHC Higgs Cross 
Section Group [TT]. Subsequent showering and hadronization effects were simulated using 
Pythia [12J. Detector effects were simulated using PGS [13] with the isolation algorithm for 
muons modified to more accurately reflect the procedure used by the CMS collaboration. 
In particular, we introduce a trkiso variable for each muon [TT]. The variable trkiso 
is defined to be the sum pt of all tracks, ECAL, and HCAL deposits within an annulus 
of inner radius 0.03 and outer radius 0.3 in AR surrounding a given muon. Isolation 
requires that for each muon, trkiso/pr is less than 0.15. The efficiencies of PGS detector 
effects were normalized by simulating the TeV3 mSUGRA benchmark studied in [B] and 
comparing the signal in 3£ and A£ channels. To match efficiencies with the CMS study 
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Table 1: Observed number of events, expected number of background events, and expected 
number of t — > ch signal events with Br(t — > ch) = 1% in various CMS multi-lepton channels 
after acceptance and efficiency for 5 fb _1 of 7 TeV proton-proton collisions. HIGH and LOW 
for MET and HT indicate $ T > 50 GeV and H T > 200 GeV respectively. DYO = £' ± £ T £ T , 
DY1 = £ ± £ + £-,f ± £ + £", for t = e,fi. No Z and Z indicate \m u - m z \ > 15 GeV for any 
opposite sign same flavor pair. 
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we applied an efficiency correction of 0.87 per lepton to our signal events pQ. We applied 
preselection and analysis cuts in accordance with those used in the CMS analysis [5J. A 
total of 500,000 events were simulated to give good statistical coverage of all the relevant 
multi-lepton channels. 

3.3 Results 

The multi-lepton final states coming from t — > ch in tt pair production arise mainly from 
charged current decay of one top quark, t — > Wb with W — > £v, and flavor-violating 
decay of the other top quark, i — > ch, with h decaying to final states with two or more 
leptons. The most relevant Higgs final states are those with two leptons that arise from 
h — > WW* — > Iviv and h — >• tt with leptonic decays of the tau-leptons, r — > EX, as well as 
h — > ZZ* — > jj££, wit. All of these decay modes give three-lepton final states. Although 
the total branching ratio of the Higgs to two leptons is comparable for h — > WW* and 
h —7- ZZ* , leptons coming from Z and/or Z* decays are less significant because they fall 
into higher-background DY1 channels with either Z or No Z . In contrast, pairs of leptons 
coming from WW* decay are uncorrelated in flavor and fall into lower-background DY0 
channels, in addition to the higher-background DY1 channels. There are additionally four- 
and five-lepton final states from charged current decay of one top quark, t — > Wb with 
W — > jj or £v respectively, and flavor-violating decay of the other top quark, i — > ch, 
with h — > ZZ* — > ££££. The small total branching ratio for these final states makes them 
less significant than the three-lepton final states in obtaining a bound from the 5 fb^ 1 of 
integrated luminosity in the CMS search [5]. However, in the future with more integrated 
luminosity, these channels should contribute more significantly to the sensitivity for t — > ch. 

The signal contributions to each of the exclusive multi-lepton channels are shown in 
Table 1. Events are entered in the table exclusive-hierarchically from the top to the bottom. 
In this way each event appears only once in the table, and in the lowest possible background 
channel consistent with its characteristics. The strongest limit-setting channels for t — > ch 
are those with three leptons. The best limits come from [MET HIGH, HT LOW, DY1 
No Z], which alone constrains Br(t ch) < 3.7%, and [MET HIGH, HT LOW, DY0], 
which constrains Br(t — > ch) < 4.2%. In each case the lack of a reconstructed Z or OSSF 
lepton pair reflects the contributions from h — > WW* and h — > tt, while the MET comes 
predominantly from neutrinos emitted in the W and r-lepton decays. Significant limits 
also come from the channel [MET HIGH, HT HIGH, DY1 No Z], which constrains Br(t — > 
ch) < 6.5%; and [MET LOW, HT LOW, DY0], which constrains Br(t ->• ch) < 7.9%; these 
likewise reflect dominant contributions from the Higgs decays h — > WW* and h — > tt. 
All other channels give constraints on the branching ratio that are weaker than 10% in an 
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individual channel. 

Although limits may be placed on the signal from any individual channel in the multi- 
lepton search, the greatest sensitivity comes from combining all exclusive channels. Com- 
bining all multilepton channels, we find that the 5 fb _1 multi-lepton CMS results j5] yield 
an observed limit of Br(£ — > ch) < 2.7%, with an expected limit Br(t — > ch) < 1.7%. This 
corresponds to a bound on the scale of the dimension-six effective operator ([2]) introduced in 
§2 of M 2 /|£ te | > (440 GeV) 2 or equivalently on the flavor- violating Higgs Yukawa coupling 
([6]) of Ajjj, < 0.22. This limit represents a combined Bayesian 95% CL limit computed using 
the observed event counts, background estimates, and systematic errors listed in Table [TJ 

An upper limit on the branching ratio Br(t — > ch) can also be expressed in terms of a 
limit on the cross section times branching ratio a-Br(pp — > ti — > Wbhc). This is related to 
the cross section and branching ratio individually by a -Br (pp —tti—t Wbhc) ~ cr(pp — > 
ti) - 2 Br(t — > he) where the factor of two accounts for combinatorics of the top quark decay. 
With this, our estimate for the observed upper limit of Br(t — > ch) < 2.7% corresponds to 
a- Br (pp —tti—> Wbhc) < 8.9 pb for 7 TeV pp collisions. While this limit is specific to 
the acceptance and efficiency associated to top-anti-top production and decay, it does give 
a rough indication of the cross section times branching limit that would be obtained from 
the results of the CMS multi-lepton search [5] for other new physics processes pp — > WhX 
with similar kinematics. 

The sensitivity of future dedicated multi-lepton searches for flavor-changing top quark 
decay t — >• ch could be improved in a number of ways. The most straightforward improve- 
ment would be to include the CMS exclusive multi-lepton channels that contain r-leptons. 
For simplicity these were neglected in this study. These channels have higher backgrounds, 
but would contribute a bit to the overall sensitivity. Another improvement would be to 
sub-divide the exclusive multi-lepton channels according to whether there are tagged b- 
quarks in an event. The t — > ch signal has both a b- and c-quark in the final state, and 
so would fall primarily in the 6-tagged channels. Although there is background from it 
production with fully-leptonic decay and a fake lepton in these channels, other Standard 
Model backgrounds from, e.g. WZ production with fully leptonic decay, would be reduced 
in these channels. Yet another possibility would be to incorporate exclusive same-sign di- 
lepton channels, again with 6-quark tagging sub-division [15]. Although the backgrounds 
in these channels are by definition larger than those of three- or more-lepton channels, this 
would bring in other relevant final states of the it signal such as charged current decay of 
one top quark, t — > Wb with W — > iu, and flavor- violating decay of the other top quark, 
i — > ch with h — > WW* — > Ivjj. Since the Higgs boson is neutral, the charges of the two 
leptons from these decays are uncorrelated and same-sign half the time. Finally, further 
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signal specific sub-divisions of channels could be utilized based on partial kinematic tagging 
information of the top quark and/or Higgs boson to isolate regions of phase space that are 
populated only by the signal. 

We emphasize that in respect to possible improvements focused at the t — > ch signal, 
the current work represents a proof of principle illustrating the power of the CMS exclusive 
channel multi-lepton search strategy [SJ E] that may be extended for certain new physics 
signals by a targeted refinement of the search channels. 

4 Conclusions 

The discovery of a Standard Model-like Higgs opens the door to a plethora of new searches 
that employ Higgs decay products to probe new physics processes that involve Higgs boson 
associated production or decay. In this paper we have studied one of the simplest such 
processes, the rare flavor-violating top quark decay to a Higgs boson and charm quark, 
t — > ch. Using the results of the CMS multi-lepton search with 5 fb _1 of 7 TeV pp collision 
data [5J, we obtain the estimated upper bounds of Br(t — > ch) < 2.7% and a -Br(pp — > ti — > 
Wbhc) < 8.9 pb for a 125 GeV Standard Model Higgs boson with Standard Model branching 
ratios. Future multi-lepton searches at the LHC optimized for this signal, including r-lepton 
channels, exclusive same-sign di-lepton channels, sub-division of channels based on 6-quark 
tagging, and with increasing integrated luminosity, should be able improve the sensitivity 
to t — )• ch considerably. 

The results presented here should be more widely applicable to a range of new physics 
processes that yield final states with a PU-boson in association with a Higgs boson. For 
processes with kinematics that are similar to top-anti-top production and decay, our esti- 
mated bound from the CMS multi-lepton search [3] corresponds very roughly to <j-Br(pp — > 
WhX) < 9 pb. Just one example of many such new physics processes that are of interest is 
production of supersymmetric wino- or Higgsino like chargino and neutralino, either directly 
or from cascade decays, with decay of the chargino to a H^-boson and lighter neutralino 
or the Goldstino, and decay of the neutralino to a Higgs boson and a lighter neutralino 
or the Goldstino, pp — > X — > x X^iX ~> WhXjXjY- I* 1 many scenarios the branching 
ratios Br(x ± — > Wx®) and Br(x° — > hx®) can approach unity [IS] . While the upper limit 
on the cross section times branching ratio obtained above does not quite bound direct 
electroweak chargino-neutralino production with these decays, it would provide bounds on 
certain scenarios with strong superpartner production where the chargino and neutralino 
are emitted in cascade decays. The future improvements to exclusive channel multi-lepton 
searches mentioned above would improve the sensitivity also to these supersymmetric pro- 
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cesses with associated Higgs bosons. In particular, direct chargino-neutralino production 
would yield final states without 6-quarks, and so would appear as signal in the 6-quark 
anti-tagged subdivision of exclusive same sign di-lepton and multi-lepton channels. 

The Higgs boson will provide a new calibration for experimental physics at high energy 
colliders. Higgs boson leptonic decay modes are but one of many possible applications of 
Higgs decays to the search for new physics. 
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